Method and apparatus for SSIM-based bit allocation

ABSTRACT

An embodiment includes a method and an encoder for SSIM-based bits allocation. The encoder includes a memory and a processor utilized for allocating bits based on SSIM, wherein the processor estimates the model parameter of SSIM-based distortion model for the current picture and determines allocates bits based on the SSIM estimation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/377,373, filed Apr. 8, 2019, which is a continuation of U.S. patentapplication Ser. No. 16/110,347, filed Aug. 23, 2018, now U.S. Pat. No.10,298,936, which is a continuation of U.S. patent application Ser. No.15/181,210, filed Jun. 13, 2016, now U.S. Pat. No. 10,085,030, which isa continuation of U.S. patent application Ser. No. 13/632,392, filedOct. 1, 2012, now U.S. Pat. No. 9,369,703, which claims the benefit ofU.S. provisional patent application Ser. No. 61/540,587, filed Sep. 29,2011, all of which are herein incorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention generally relate to a method andapparatus for SSIM-based bit allocation.

Description of the Related Art

Bit rate affects the video quality. Thus, it is crucial to allocate theeffective amount of bits per frame to maintain quality andefficiency/cost. Mean Square Error is still major metric being used invideo encoder control and optimization. However, Mean Square Error basedencoder is far from perceptual optimization. Even though SSIM(Structural Similarity) index is a good quality metric for subjectivevideo quality assessment and more correlated than Mean Square Error to ahuman's visual perception, yet, currently, there is no SSIM-based rateand/or distortion models.

Therefore, there is a need for a method and/or apparatus for SSIM-basedbit allocation.

SUMMARY OF THE INVENTION

Embodiments of the present invention relate to a method and an encoderfor SSIM-based bits allocation. The encoder includes a memory and aprocessor utilized for allocating bits based on SSIM, wherein theprocessor estimates the parameter of SSIM-based distortion model for thecurrent picture and determines allocates bits based on the SSIMestimation.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is an embodiment of a relationship between ln (D′ssim/Dssim) andRate;

FIG. 2 is an embodiment of a flow diagram depicting a method forSSIM-based bit allocation; and

FIG. 3 is an embodiment of an encoder.

DETAILED DESCRIPTION

The proposed invention minimizes overall SSIM distortion, which is morecorrelated with human perceptual quality than MSE, while existing bitallocation methods focus on minimization of overall MSE distortion

The proposed invention provides the optimal number of bits for eachcoding unit in a closed form. And encoded video by the proposed bitallocation will be more pleasing to human visual system.

SSIM index evaluates the quality of reconstructed coded frame r bycomparing luminance, contrast and structural similarities between r andoriginal frame o. That is,SSIM (o,r)=l(o,r)·c(o,r)·s(o,r),where

${{l\left( {o,r} \right)} = \frac{{2\mu_{o}\mu_{r}} + C_{1}}{\mu_{o}^{2} + \mu_{r}^{2} + C_{1}}},{{c\left( {o,r} \right)} = {\frac{{2\sigma_{o}\sigma_{r}} + C_{2}}{\sigma_{o}^{2} + \sigma_{r}^{2} + C_{2}}\mspace{14mu}{and}}}$${s\left( {o,r} \right)} = {\frac{{2\sigma_{or}} + C_{3}}{{\sigma_{o}\sigma_{r}} + C_{3}}.}$C₁, C₂ and C₃ are constants to avoid unstable behavior in the regions oflow luminance or low contrast.

The range of SSIM index is 0 to 1. SSIM index is close to 1 when twoframes are similar. For example, when two frames are identical, SSIMindex is 1. So distortion is 1−SSIM. With MSE as a distortion metric, itis well known that distortion is modeled byD _(MSE)=σ²·exp{−βR),where σ² is variance of residual signal and where β is model parameter.Residual signal is difference between original and prediction. Weobserve that distortion in terms of SSIM (i.e. 1−SSIM) is modeled by thesimilar function. That is,1−SSIM (o,r)=(1−SSIM (o,p))·exp{−βR),where is model parameter and p is prediction.By replacing 1−SSIM (o, r) and 1−SSIM (o, p) with D_(ssim) andD′_(ssim), respectively, for simplicity, we haveD _(SSIM) =D′ _(SSIM)·exp(−βR),  (1)where is model parameter and p is prediction.

FIG. 1 is an embodiment of a relationship between ln (D′ssim/Dssim) andRate. FIG. 1 shows the relationship between ln (D′_(SSIM)/D_(SSIM)) andrate for 5 consecutive P frames from 4 720p sequences. Hence, Eq. (1) isvalid with different values of β depending on the characteristics offrames (coding units).

Applying the SSIM-based distortion model for perceptually optimized bitallocation and assuming that there are n coding units (e.g. frame) toencode with total bit budge R_(T), the overall perceptual quality isoptimized with R_(T). That is,

${{minimize}\mspace{14mu}{\sum\limits_{i = 1}^{N}{D_{{SSIM},i}\mspace{14mu}{subject}\mspace{14mu}{to}\mspace{14mu}{\sum\limits_{i = 1}^{N}R_{i}}}}} \leq {R_{T}.}$

Here we assume that all coding units are independent. This constrainedproblem can be converted to the unconstrained problem with Lagrangemultiplier:

$\begin{matrix}{{{minimize}\mspace{14mu} J} = {{\sum\limits_{i = 1}^{N}D_{{SSIM},i}} + {\lambda \cdot \left( {{\sum\limits_{i = 1}^{N}R_{i}} - R_{T}} \right)}}} \\{= {{\sum\limits_{i = 1}^{N}{\beta_{i} \cdot D_{{SSIM},i}^{\prime} \cdot {\exp\left( {{- \beta_{i}}R_{i}} \right)}}} + {\lambda \cdot {\left( {{\sum\limits_{i = 1}^{N}R_{i}} - R_{T}} \right).}}}}\end{matrix}$

By setting partial derivative_w.r.t and λ and R_(k) to 0, we haveoptimal bits for coding unit k as

$\begin{matrix}{R_{k} = {{\frac{1}{\beta_{k}}{\ln\left( {\beta_{k}\  \cdot \ D_{{SSIM},k}^{\prime}} \right)}} + {\frac{1}{\beta_{k} \cdot {\sum\limits_{i = 1}^{N}\frac{1}{\beta_{i}}}} \cdot {\left\{ {R_{T} - {\sum\limits_{i = 1}^{N}{\frac{1}{\beta_{i}}{\ln\left( {\beta_{i}\ .\ D_{{SSIM},i}^{\prime}} \right)}}}} \right\}.}}}} & (2)\end{matrix}$

The proposed bit allocation in Eq. 2 can be implemented in various ways.For example, two-pass method and approximated one-pass method. In twopass method, all coding units are coded with fixed QP in the first passto get β and D′_(SSIM) for all coding units in consideration. Then afterdetermining R_(k) for all coding units, they are finally coded in thesecond pass. In the approximated one-pass method, β and D′_(SSIM) areapproximated from previous coding units. In case of frame bit allocationin GOP, β and D′_(SSIM) values of all frames in a GOP can beapproximated from frames at the same positions in the previous GOP.

FIG. 2 is an embodiment of a flow diagram depicting a method 2100 forSSIM-based bit allocation. The method 200 starts at step 202 andproceeds to step 204. At step 204, the method 200 set GOP count to zero.At step 206, the method 200 increments the GOP count by 1. At step 208,the method 200 sets the picture count to zero. At step 210, the method200 encodes the current picture in the current GOP. At step 212, themethod 200 estimates the model parameter of SSIM-based distortion modelfor the current picture. At step 214, the method 200 determines if thecurrent picture is the last picture in the current GOP.

If it is not the last picture in the current GOP, the method 200proceeds to step 216, wherein the method 200 increments the picturecount and returns to step 206; otherwise the method proceeds to step218. At step 218, the method 200 determines if the current GOP is thelast GOP. If it is, the method 200 proceeds to step 222; otherwise, themethod 200 proceeds to step 220. At step 220, the method 200 determinesthe target bits for each frame for the next GOP and returns to step 210.The method 200 ends at step 222.

FIG. 3 shows a block diagram of the largest coding units (LCU)processing portion of an example video encoder. A coding controlcomponent (not shown) sequences the various operations of the LCUprocessing, i.e., the coding control component runs the main controlloop for video encoding. The coding control component receives a digitalvideo sequence and performs any processing on the input video sequencethat is to be done at the picture level, such as determining the codingtype (I, P, or B) of a picture based on the high level coding structure,e.g., IPPP, IBBP, hierarchical-B, and dividing a picture into LCUs forfurther processing. The coding control component also may determine theinitial LCU coding unit (CU) structure for each CU and providesinformation regarding this initial LCU CU structure to the variouscomponents of the video encoder as needed. The coding control componentalso may determine the initial prediction unit (PU) and transform unit(TU) structure for each CU and provides information regarding thisinitial structure to the various components of the video encoder asneeded.

The LCU processing receives LCUs of the input video sequence from thecoding control component and encodes the LCUs under the control of thecoding control component to generate the compressed video stream. TheCUs in the CU structure of an LCU may be processed by the LCU processingin a depth-first Z-scan order. The LCUs 300 from the coding control unitare provided as one input of a motion estimation component 320, as oneinput of an intra-prediction component 324, and to a positive input of acombiner 302 (e.g., adder or subtractor or the like). Further, althoughnot specifically shown, the prediction mode of each picture as selectedby the coding control component is provided to a mode selector componentand the entropy encoder 334.

The storage component 318 provides reference data to the motionestimation component 320 and to the motion compensation component 322.The reference data may include one or more previously encoded anddecoded CUs, i.e., reconstructed CUs.

The motion estimation component 320 provides motion data information tothe motion compensation component 322 and the entropy encoder 334. Morespecifically, the motion estimation component 320 performs tests on CUsin an LCU based on multiple inter-prediction modes (e.g., skip mode,merge mode, and normal or direct inter-prediction) and transform blocksizes using reference picture data from storage 318 to choose the bestmotion vector(s)/prediction mode based on a rate distortion coding cost.To perform the tests, the motion estimation component 320 may begin withthe CU structure provided by the coding control component. The motionestimation component 320 may divide each CU indicated in the CUstructure into PUs according to the unit sizes of prediction modes andinto transform units according to the transform block sizes andcalculate the coding costs for each prediction mode and transform blocksize for each CU. The motion estimation component 320 may also computeCU structure for the LCU and PU/TU partitioning structure for a CU ofthe LCU by itself.

For coding efficiency, the motion estimation component 320 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best motionvectors/prediction modes, in addition to testing with the initial CUstructure, the motion estimation component 320 may also choose to dividethe larger CUs in the initial CU structure into smaller CUs (within thelimits of the recursive quadtree structure), and calculate coding costsat lower levels in the coding hierarchy. If the motion estimationcomponent 320 changes the initial CU structure, the modified CUstructure is communicated to other components that need the information.

The motion estimation component 320 provides the selected motion vector(MV) or vectors and the selected prediction mode for eachinter-predicted PU of a CU to the motion compensation component 322 andthe selected motion vector (MV), reference picture index (indices),prediction direction (if any) to the entropy encoder 334

The motion compensation component 322 provides motion compensatedinter-prediction information to the mode decision component 326 thatincludes motion compensated inter-predicted PUs, the selectedinter-prediction modes for the inter-predicted PUs, and correspondingtransform block sizes. The coding costs of the inter-predicted PUs arealso provided to the mode decision component 326.

The intra-prediction component 324 provides intra-prediction informationto the mode decision component 326 that includes intra-predicted PUs andthe corresponding intra-prediction modes. That is, the intra-predictioncomponent 324 performs intra-prediction in which tests based on multipleintra-prediction modes and transform unit sizes are performed on CUs inan LCU using previously encoded neighboring PUs from the buffer 328 tochoose the best intra-prediction mode for each PU in the CU based on acoding cost.

To perform the tests, the intra-prediction component 324 may begin withthe CU structure provided by the coding control. The intra-predictioncomponent 324 may divide each CU indicated in the CU structure into PUsaccording to the unit sizes of the intra-prediction modes and intotransform units according to the transform block sizes and calculate thecoding costs for each prediction mode and transform block size for eachPU. For coding efficiency, the intra-prediction component 324 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best predictionmodes, in addition to testing with the initial CU structure, theintra-prediction component 324 may also chose to divide the larger CUsin the initial CU structure into smaller CUs (within the limits of therecursive quadtree structure), and calculate coding costs at lowerlevels in the coding hierarchy. If the intra-prediction component 324changes the initial CU structure, the modified CU structure iscommunicated to other components that need the information. Further, thecoding costs of the intra-predicted PUs and the associated transformblock sizes are also provided to the mode decision component 326.

The mode decision component 326 selects between the motion-compensatedinter-predicted PUs from the motion compensation component 322 and theintra-predicted PUs from the intra-prediction component 324 based on thecoding costs of the PUs and the picture prediction mode provided by themode selector component. The decision is made at CU level. Based on thedecision as to whether a CU is to be intra- or inter-coded, theintra-predicted PUs or inter-predicted PUs are selected, accordingly.

The output of the mode decision component 326, i.e., the predicted PU,is provided to a negative input of the combiner 302 and to a delaycomponent 330. The associated transform block size is also provided tothe transform component 304. The output of the delay component 330 isprovided to another combiner (i.e., an adder) 338. The combiner 302subtracts the predicted PU from the current PU to provide a residual PUto the transform component 304. The resulting residual PU is a set ofpixel difference values that quantify differences between pixel valuesof the original PU and the predicted PU. The residual blocks of all thePUs of a CU form a residual CU block for the transform component 304.

The transform component 304 performs block transforms on the residual CUto convert the residual pixel values to transform coefficients andprovides the transform coefficients to a quantize component 306. Thetransform component 304 receives the transform block sizes for theresidual CU and applies transforms of the specified sizes to the CU togenerate transform coefficients.

The quantize component 306 quantizes the transform coefficients based onquantization parameters (QPs) and quantization matrices provided by thecoding control component and the transform sizes. The quantize component306 may also determine the position of the last non-zero coefficient ina TU according to the scan pattern type for the TU and provide thecoordinates of this position to the entropy encoder 334 for inclusion inthe encoded bit stream. For example, the quantize component 306 may scanthe transform coefficients according to the scan pattern type to performthe quantization, and determine the position of the last non-zerocoefficient during the scanning/quantization.

The quantized transform coefficients are taken out of their scanordering by a scan component 308 and arranged sequentially for entropycoding. The scan component 308 scans the coefficients from the highestfrequency position to the lowest frequency position according to thescan pattern type for each TU. In essence, the scan component 308 scansbackward through the coefficients of the transform block to serializethe coefficients for entropy coding. As was previously mentioned, alarge region of a transform block in the higher frequencies is typicallyzero. The scan component 308 does not send such large regions of zerosin transform blocks for entropy coding. Rather, the scan component 308starts with the highest frequency position in the transform block andscans the coefficients backward in highest to lowest frequency orderuntil a coefficient with a non-zero value is located. Once the firstcoefficient with a non-zero value is located, that coefficient and allremaining coefficient values following the coefficient in the highest tolowest frequency scan order are serialized and passed to the entropyencoder 334. In some embodiments, the scan component 308 may beginscanning at the position of the last non-zero coefficient in the TU asdetermined by the quantize component 306, rather than at the highestfrequency position.

The ordered quantized transform coefficients for a CU provided via thescan component 308 along with header information for the CU are coded bythe entropy encoder 334, which provides a compressed bit stream to avideo buffer 336 for transmission or storage. The header information mayinclude the prediction mode used for the CU. The entropy encoder 334also encodes the CU and PU structure of each LCU.

The LCU processing includes an embedded decoder. As any compliantdecoder is expected to reconstruct an image from a compressed bitstream, the embedded decoder provides the same utility to the videoencoder. Knowledge of the reconstructed input allows the video encoderto transmit the appropriate residual energy to compose subsequentpictures. To determine the reconstructed input, i.e., reference data,the ordered quantized transform coefficients for a CU provided via thescan component 308 are returned to their original post-transformarrangement by an inverse scan component 310, the output of which isprovided to a dequantize component 312, which outputs a reconstructedversion of the transform result from the transform component 304.

The dequantized transform coefficients are provided to the inversetransform component 314, which outputs estimated residual informationwhich represents a reconstructed version of a residual CU. The inversetransform component 314 receives the transform block size used togenerate the transform coefficients and applies inverse transform(s) ofthe specified size to the transform coefficients to reconstruct theresidual values. The inverse transform component 314 may performtechniques for IDCT pruning as described herein.

The reconstructed residual CU is provided to the combiner 338. Thecombiner 338 adds the delayed selected CU to the reconstructed residualCU to generate an unfiltered reconstructed CU, which becomes part ofreconstructed picture information. The reconstructed picture informationis provided via a buffer 328 to the intra-prediction component 324 andto an in-loop filter component 316. The in-loop filter component 316applies various filters to the reconstructed picture information toimprove the reference picture used for encoding/decoding of subsequentpictures. The in-loop filter component 316 may, for example, adaptivelyapply low-pass filters to block boundaries according to the boundarystrength to alleviate blocking artifacts causes by the block-based videocoding. The filtered reference data is provided to storage component318.

The encoder efficiency to perform these functions is largely dependenton bit allocation. The encoder 300 allocated bits based on SSIMestimations. Such allocation is described in more detail in FIG. 2.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

The invention claimed is:
 1. An encoder, comprising: a coding controlcomponent, configured to: receive a digital video frame; and divide thedigital video frame onto a plurality of coding units (CUs); a transformcomponent coupled to the coding control component, the transformcomponent configured to perform block transforms on the plurality ofCUs, to generate a plurality of transform coefficients; and a quantizecomponent coupled to the transform component, the quantize componentconfigured to: determine a plurality of parameters of a structuralsimilarity index (SSIM)-based distortion model for the plurality of CUs,based on characteristics of the plurality of CUs; allocate a pluralityof bit budgets for the plurality of coding units based on the pluralityof parameters; and quantize the plurality of transform coefficientsusing the plurality of bit budgets, to generate a plurality of quantizedcoefficients.
 2. The encoder of claim 1, further comprising a scancomponent coupled to the quantize component, the scan componentconfigured to sequentially arrange the plurality of quantizedcoefficients, to generate a plurality of ordered transform coefficients.3. The encoder of claim 2, further comprising an entropy encoder coupledto the scan component, the entropy encoder configured to: performentropy encoding on the plurality of ordered transform coefficients, togenerate a compressed bit stream; and store the compressed bit stream ina video buffer.
 4. The encoder of claim 3, further comprising atransmitter coupled to the video buffer, the transmitter configured totransmit the compressed bit stream.
 5. The encoder of claim 3, furthercomprising storage coupled to the video buffer, the storage configuredto store the compressed bit stream.
 6. The encoder of claim 2, furthercomprising: an inverse scan component coupled to the scan component, theinverse scan component configured to arrange the plurality of orderedtransform coefficients, to generate a plurality of reordered transformcoefficients; a dequantize component coupled to the inverse scancomponent, the dequantize component configured to dequantize theplurality of reordered transform coefficients, to generate a pluralityof reconstructed transform results; and an inverse transform componentcoupled to the dequantize component, the inverse transform componentconfigured to: generate a second plurality of transform coefficientsbased on the plurality of reconstructed transform results; and apply aninverse transform to the second plurality of transform coefficients, togenerate a plurality of reconstructed residual values.
 7. The encoder ofclaim 6, further comprising an in-loop filter component coupled to theinverse transform component, the in-loop filter component configured to:filter the plurality of reconstructed residual values, to generatefiltered reference data; and store the filtered reference data instorage.
 8. The encoder of claim 1, wherein the quantize component isfurther configured to: perform a first encoding of the plurality of CUswith fixed quantization parameters, to generate first pass coding;determine the plurality of parameters for the plurality of CUs based onthe first pass coding; and determine the plurality of bit budgets forthe plurality of CUs based on the plurality of parameters.
 9. Theencoder of claim 8, wherein the quantize component is further configuredto perform a second encoding of the plurality of CUs based on theplurality of bit budgets.
 10. The encoder of claim 1, wherein thequantize component is further configured to determine the plurality ofparameters for the plurality of CUs is performed based on previouscoding units.
 11. A decoder, comprising: a dequantize componentconfigured to dequantize a plurality of transform coefficients, togenerate a plurality of reconstructed transform results, wherein theplurality of transform coefficients have been generated by an encoderconfigured to: determine a plurality of parameters of a structuralsimilarity index (SSIM)-based distortion model for the plurality of CUs,based on characteristics of the plurality of CUs; allocate a pluralityof bit budgets for the plurality of coding units based on the pluralityof parameters; and quantize the plurality of transform coefficientsusing the plurality of bit budgets, to generate a plurality of quantizedcoefficients; and an inverse transform component coupled to thedequantize component, the inverse transform component configured to:generate a second plurality of transform coefficients based on theplurality of reconstructed transform results; and apply an inversetransform to the second plurality of transform coefficients, to generatea reconstructed picture.
 12. The decoder of claim 11, further comprisingan inverse scan component coupled to the dequantize component, theinverse scan component configured to arrange a plurality of orderedtransform coefficients, to generate the plurality of transformcoefficients.
 13. The decoder of claim 11, further comprising a receivercoupled to the dequantize component, the receiver configured to receivethe plurality of transform coefficients.
 14. The decoder of claim 11,wherein the decoder is configured to retrieve the plurality of transformcoefficients from storage.
 15. The decoder of claim 11, wherein thedecoder is configured to send the reconstructed picture to a display.16. The decoder of claim 11, wherein the decoder is configured to storethe reconstructed picture in storage.
 17. A method comprising:receiving, by a coding control component of an encoder, a digital videoframe; and dividing, by the coding control component of the encoder, thedigital video frame onto a plurality of coding units (CUs); performing,by a transform component of the encoder, block transforms on theplurality of CUs, to generate a plurality of transform coefficients;determining, by a quantize component of the encoder, a plurality ofparameters of a structural similarity index (SSIM)-based distortionmodel for the plurality of CUs, based on characteristics of theplurality of CUs; allocating, by the quantize component of the encoder,a plurality of bit budgets for the plurality of coding units based onthe plurality of parameters; and quantizing, by the quantize componentof the encoder, the plurality of transform coefficients using theplurality of bit budgets, to generate a plurality of quantized transformcoefficients.